Omnivergent stereo image capture

ABSTRACT

Methods and apparatuses for obtaining a first image and a second image defining a convergent stereo image, and constructing a three-dimensional images based on the obtained first and second images. In one embodiment, a deflector, such as a prism, is rotably mounted about an axis, and first and second inputs received by the deflector are deflected towards a receptor at or proximate to the axis. In one embodiment, the axis and the deflector are perpendicular to a rotation plane, and the first and second inputs are tangential to a region defined by the rotation of the deflector about the axis, opposing each other, and parallel to the rotation plane. The first and second images are respectively determined with respect to first and second inputs received as the deflector rotates about the axis.

FIELD OF THE INVENTION

[0001] The invention generally relates to image capture, and moreparticularly to capturing omnivergent stereo image pairs for compactlyrepresenting 360° panoramic images of an environment surrounding acapture device.

BACKGROUND

[0002] There are many known techniques for generating three-dimensionalimages of a surrounding environment. Methods referred to as “360panorama”, “Panoramic stereo imaging” and “omnivergent imaging” concerndistinct techniques for creating such three-dimensional images. Sometechniques utilize one or more image capturing devices, e.g., cameras orother input source, to define stereo images corresponding to anenvironment about the image capturing devices.

[0003] For example, the conference paper Stereo panorama with a singlecamera, Peleg and Ben-Ezra, in Proc. Computer Vision and PatternRecognition Conf., pp. 395-401 (1999), discusses creating mosaic imagesfrom a rotating camera, dealing with incident problems of parallax andscale changes, and using a single camera to create two stereo panoramicviews, one for each eye, through multiple viewpoint image projections.

[0004] The paper Stereo reconstruction from multiperspective panoramas,Heung-Yeung Shum and Richard Szeliski, in IEEE Int'l Conference onComputer Vision, pp. 14-21 vol. 1 (1999), discusses computing depth mapsfrom a collection of images, where camera motion is constrained toplanar concentric circles. The resulting collection of regularperspective images is sampled into a set of multiple perspectivepanoramas, and depth maps can be computed from the sampled images.

[0005]The conference paper Omnivergent Stereo by Heung-Yeung Shum, andSeitz, in IEEE Int'l Conference on Computer Vision, pp. 22-29 vol. 1(1999), discusses a virtual sensor for 3D image construction, whereinstead of using planar perspective images that collect many rays at afixed viewpoint, omnivergent cameras are instead used to collect a smallnumber of selected rays at many different viewpoints. The collected 2Dmanifold of rays is arranged into two multiple-perspective imagesallowing for stereo reconstruction, and, according to the authors, 3Dmodels can be reconstructed with minimal error since every point in themodel is defined with respect to two rays having maximum vergence angle.However, this document fails to teach how a physical capture device canbe created to implement the techniques discussed in the document.

[0006] It will be appreciated by those skilled in the art that thesethree references are presented for exemplary purposes to illustratecurrent state of the art, and to show lack of knowledge in the art as tohow to effectively build an omnivergent stereo image capture device notsuffering from limitations discussed in these references.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The features and advantages of the present invention will becomeapparent from the following detailed description of the presentinvention in which:

[0008]FIG. 1 illustrates a top view according to one embodiment of animage recorder having a cylindrical region defined by the rotation pathof a deflector.

[0009]FIG. 2 illustrates a rotation of the FIG. 1 deflector.

[0010]FIG. 3 illustrates an alternate embodiment utilizing fourdeflectors and four sensors.

[0011]FIG. 4 illustrates a perspective view of the FIG. 1 embodiment.

[0012]FIG. 5 illustrates a suitable computing environment in whichcertain aspects of the invention may be implemented.

DETAILED DESCRIPTION

[0013] The following detailed description assumes reader familiaritywith the mathematics and principles of stereo and omnivergent imaging.

[0014]FIG. 1 illustrates a top view according to one embodiment of anomnivergent stereo recorder. A cylindrical region 100 is defined by therotation path of a deflector 102 about an axis of rotation 104 that isperpendicular to a rotation plane 106. In one embodiment, on, adjacentor proximate to the axis of rotation is a receptor 108 comprising anarray of image sensors that is also perpendicular to the rotation plane.In illustrated embodiments, the receptor comprises two one-dimensionalcolumns 110, 112 of sensors. However, it will be appreciated that otherarrangements may be used.

[0015] As illustrated, there are two inputs 114, 116 to the deflector102 that are tangential to the cylindrical region 100. In oneembodiment, the inputs are directly opposing each other so as to form a180° angle between them. In one embodiment, the deflector is a prism andthe inputs are light rays. It will be appreciated, however, othervisible and/or non-visible electromagnetic energy may be converged withthe deflector 102, and the deflector will have properties appropriate tothe electromagnetic energy being reflected. For example, the deflectormay be a physical deflector—in the case of a prism deflecting light. Forcertain other forms of electromagnetic energy, the deflector may be agenerated deflection field, such as a magnetic field.

[0016] When the two inputs 114, 116 reach the deflector 102, they aredeflected so as to converge on the sensors 110, 112. Assuming aleft-to-right arrangement in the figure, as illustrated, the left input114 is deflected towards the left sensor 110, and the right input 116 isdeflected towards the right sensor 112. For each rotational position ofthe deflector-sensor assembly, the sensors are used to record thereceived input for the given rotational position. After recording theinput, as illustrated in FIG. 2, the sensors can be rotated 200 to a newrotational position and inputs collected again. This process ofrotation, collection of input, and continuing rotation can be repeatedcontinuously.

[0017] The rate of rotation can be accelerated or decreased according totemporal sampling needs. For example, to generate a live broadcast ofthree-dimensional data the rotation speed needs to be at least 1800rotations per minute (RPM) to maintain a frame rate of 30 frames persecond (FPS). It will be appreciated that a 30 FPS can also be achievedwith a lower rotational speed by having multiple deflectors 102 andreceptors 108. For example, as illustrated in FIG. 3, four deflectors300 can be arranged with four sensors 302 to allow four concurrentsamples to be taken for a given rotation position. This embodimentreduces rotation speed to 450 RPM for maintaining a frame rate of 30FPS, since a full revolution of samples is now collected in a quarterturn. Alternatively, this embodiment allows quadrupling the frame rate.It will be appreciated that an arbitrary number of sensors anddeflectors may be used.

[0018] To create an omnivergent stereo image pair, a first image isdefined by compiling all leftward inputs, i.e. the set of inputs 114,captured over a complete revolution of a deflector-sensor assembly(e.g., a combination of a deflector 102 and a sensor 108) A second imageis similarly defined using the rightward inputs, i.e. the set of inputs116 captured over a complete revolution of the deflector-sensorassembly. These two images form an omnivergent stereo image pair whichcompactly represents the three dimensional structure of the environmentsurrounding the invention.

[0019]FIG. 4 illustrates a perspective view of the FIG. 1 embodiment. Asin FIG. 1, illustrated are the cylindrical region 100 defined by therotation path of the deflector 102 about the axis of rotation 104. Ascan now be seen in FIG. 4, the deflector and sensors are mechanicallyrelated. As the deflector is rotated, the array of sensors 400, 402(corresponding to FIG. 1 items 110, 112) rotates in tandem with thedeflector. During rotation, the sensors constantly receive the deflectedinput 114, 116.

[0020] It will be appreciated that sensor arrays 400, 402, can bearbitrarily dense and have diverse physical arrangements depending onthe technology used to create the sensor arrays. The more sensors thereare, the better the vertical resolution of captured images, and whencombined with arbitrary rotation speeds, captured image data can bearbitrarily detailed and dense. It will also be appreciated that thedistance between the deflector 102 and the axis of rotation 104 may bearbitrarily adjusted so as to determine a desired depth of field andmaximum vergence angle for imaging a particular environment.

[0021] Once the omnivergent stereo images have been captured anddetermined, one can then select an arbitrary viewpoint from within thecylindrical region 100 and use image-based rendering techniques torender a virtual image with respect to the arbitrarily selectedviewpoint. Selection and rendering may be performed in real time,allowing for significant advances in telepresence, video conferencingapplications, three dimensional gaming applications, and otherapplications in which it would be useful to allow users to selectarbitrary viewpoints in a scene. The invention enables users toexperience holographic television without the need for holograms asintermediate storage. The limitation is that the viewpoints can only beselected from within the D cylinder 100.

[0022] In one embodiment, multiple omnivergent stereo image capturingdevices (not illustrated) are used to determine omnivergent stereoimages from multiple viewpoints. These multiple omnivergent stereoimages are then synthesized into a large viewpoint selection regioncomprising the individual cylindrical regions of the multiple capturingdevices. In one embodiment, the multiple capturing devices are arrangedso that their cylindrical regions abut each other. In anotherembodiment, synthesizing includes computing values for interveningspaces between cylindrical regions.

[0023]FIG. 5 and the following discussion are intended to provide abrief, general description of a suitable computing environment in whichcertain aspects of the illustrated invention may be implemented. It willbe appreciated by one skilled in the art that the invention may be usedin applications such as simulated holographic televisionvideoconferencing, three-dimensional gaming, virtual realityenvironment, to capture and/or react to movement within an environment,e.g., user gesticulation, or other application desiringthree-dimensional representations of an environment.

[0024] Towards these ends, the invention may incorporate or beincorporated within, controlled by, or otherwise communicatively coupledwith a machine 500 having system bus 502 for coupling various machinecomponents. Typically, attached to the bus are one or more processors504, a memory 506 (e.g., RAM, ROM), storage devices 508, a videointerface 510, and input/output interface ports 512. The system mayinclude embedded controllers, such as programmable logic devices or gatearrays, Application Specific Integrated Circuits (ASIC), single-chipcomputers, etc.

[0025] The system may operate in a networked environment using physicaland/or logical connections to one or more remote systems 514, 516through a network interface 518, modem 520, or other pathway. Systemsmay be interconnected by way of a wired or wireless network 522,including an intranet, the Internet, local area networks, wide areanetworks, cellular, cable, laser, satellite, microwave, “Blue Tooth”type networks, optical, infrared, or other carrier.

[0026] The invention may be described by reference to program modules orinstructions for performing tasks or implementing abstract data types,e.g., procedures, functions, data structures, application programs,etc., that may be stored in memory 506 and/or storage devices 508 andassociated volatile and non-volatile storage media, e.g., magnetic,optical, biological, or other storage, as well as transmissionenvironments such as network 522 over which program modules may bedelivered in the form of packets, serial data, parallel data, or othertransmission format. Program modules or instructions may be stored inmachine accessible media, including wired and wirelessly accessiblemedia.

[0027] Thus, for example, assuming a three-dimensional videoconferencingor multi-player game, machine 500 and remote devices 514, 516 operate todetermine an omnivergent stereo image of their respective environments.These stereo images can then be shared among users of the machines 500,514, 516 to allow respective users to selectively define viewpointswithin other users' environments. It will be appreciated that remotemachines 514, 516 may be configured like machine 500, and thereforeinclude many or all of the elements discussed for machine. It shouldalso be appreciated that machines 500, 514, 516 may be embodied within asingle device, or separate communicatively-coupled components.

[0028] Having described and illustrated the principles of the inventionwith reference to illustrated embodiments, it will be recognized thatthe illustrated embodiments can be modified in arrangement and detailwithout departing from such principles. And, even though the foregoingdiscussion has focused on particular embodiments, it is understood otherconfigurations are contemplated. In particular, even though expressionssuch as “in one embodiment,” “in another embodiment,” or the like areused herein, these phrases are meant to generally reference embodimentpossibilities, and are not intended to limit the invention to particularembodiment configurations. As used herein, these terms may reference thesame or different embodiments, and unless indicated otherwise,embodiments are combinable into other embodiments.

[0029] Consequently, in view of the wide variety of permutations to theabove-described embodiments, the detailed description is intended to beillustrative only, and should not be taken as limiting the scope of theinvention. What is claimed as the invention, therefore, is all suchmodifications as may come within the scope and spirit of the followingclaims and equivalents thereto.

1. A method for constructing a first image and a second image of anomnivergent stereo image pair, comprising: rotating a deflector about arotation axis, the deflector positioned a distance from the rotationaxis and having plural deflection regions; positioning a receptorproximate to the rotation axis, the receptor comprising a first portionof sensors and a second portion of sensors; deflecting a first inputreceived at a first deflection region of the deflector to the firstportion of sensors; deflecting a second input received at a seconddeflection region of the deflector to a second portion of sensors;determining the first image based at least in part on the first input;determining the second image based at least in part on the second input;and determining a first omnivergent stereo pair based at least in parton the first image and the second image:
 2. The method of claim 1,further comprising: wherein both the first image and the second imageare omnivergent images.
 3. The method of claim 1, further comprising:selecting a view point; and rendering a three dimensional imaged basedat least in part on the view point and the first omnivergent stereopair.
 4. The method of claim 1, wherein the distance is fixed.
 5. Themethod of claim 1, further comprising: performing the method at a firstlocation to determine the first omnivergent stereo pair; performing themethod at a second location to determine a second omnivergent stereopair; and synthesizing an environment model based at least in part onthe first omnivergent stereo pair and the second omnivergent stereopair.
 6. The method of claim 5, wherein the first location is proximateto the second location.
 7. The method of claim 5, wherein a first regiondefined by rotating the deflector about the axis at the first locationabuts a second region defined by rotating the deflector about the axisat the second location.
 8. The method of claim 1, further comprising:receiving a configuration input; and setting the distance with respectto the configuration input.
 9. The method of claim 8, wherein theconfiguration input corresponds to a desired size for a region in whicha viewpoint may be selected.
 10. The method of claim 9, furthercomprising: receiving a viewpoint selection; and rendering a threedimensional image based on the viewpoint selection and the first and thesecond image.
 11. A method for constructing an omnivergent stereo imagepair, comprising: defining a cylindrical region having an axis ofrotation perpendicular to a rotation plane, the cylindrical regiondefined with respect to an array of sensors disposed parallel to theaxis of rotation, and a prism disposed parallel to the vertical array;and determining an environment about the cylindrical region by rotatingthe cylindrical region through rotational positions, and while rotating:receiving a first input at a first face of the prism for a rotationalposition of the cylindrical region, the first input having a firsttravel path tangential to the cylindrical region and corresponding to afirst portion of the environment, and receiving a second input at asecond face of the prism for the rotational position of the cylindricalregion, the second input having a second travel path tangential to thecylindrical region and corresponding to a second portion of theenvironment.
 12. The method of claim 9, further comprising: storing thefirst input and the second input for each of plural rotational positionsof the cylindrical region; selecting a view point within the cylindricalregion; and constructing a convergent stereo image of the environmentwith respect to the selected view point and the stored first and secondinputs for the plural rotational positions of the cylindrical region.11. The method of claim 9, wherein the first travel path is opposite ofthe second travel path.
 12. The method of claim 9, wherein the first andsecond travel paths are parallel to the rotation plane.
 13. An articleof manufacture, comprising: a machine accessible medium havingassociated data, which when accessed by the machine, results in themachine performing: rotating a deflector rotably mounted a distance froma rotation axis, the deflector having plural deflection regions fordeflecting inputs to a receptor positioned proximate to the rotationaxis, the receptor comprising a first portion of sensors and a secondportion of sensors; determining the first image based at least in parton a first input received at a first deflection region of the deflectorthat is deflected towards the receptor; determining the second imagebased at least in part on a second input received at a second deflectionregion of the deflector that is deflected towards the receptor;determining a first omnivergent stereo pair based at least in part onthe first image and the second image.
 14. The apparatus of claim 13,wherein both the first image and the second image are omnivergentimages.
 15. The apparatus of claim 13, QQQ: selecting a view point; andrendering a three dimensional imaged based at least in part on the viewpoint and the first omvivergent stereo pair.
 16. The apparatus of claim13, wherein the distance is fixed.
 17. The apparatus of claim 13, QQQperforming the method at a first location to determine the firstomnivergent stereo pair; performing the method at a second location todetermine a second omnivergent stereo pair; and synthesizing anenvironment model based at least in part on the first omnivergent stereopair and the second omnivergent stereo pair.
 18. The apparatus of claim17, wherein the first location is proximate to the second location. 19.The apparatus of claim 17, wherein a first region defined by rotatingthe deflector about the axis at the first location abuts a second regiondefined by rotating the deflector about the axis at the second location.20. The apparatus of claim 13, QQQ: receiving a configuration input; andsetting the distance with respect to the configuration input.
 21. Theapparatus of claim 20, wherein the configuration input corresponds to adesired size for a region in which a viewpoint may be selected.
 22. Theapparatus of claim 21, QQQ: receiving a viewpoint selection; andrendering a three dimensional image based on the viewpoint selection andthe first and the second image.
 23. An apparatus comprising a machineaccessible medium having instructions associated therewith forconstructing a first image and a second image of a convergent stereoimage pair, the instructions capable of directing a machine to perform:defining a cylindrical region having an axis of rotation perpendicularto a rotation plane, the cylindrical region defined with respect to anarray of sensors disposed parallel to the axis of rotation, and a prismdisposed parallel to the vertical array; determining an environmentabout the cylindrical region by rotating the cylindrical region throughrotational positions, and while rotating: receiving a first input at afirst face of the prism for a rotational position of the cylindricalregion, the first input having a first travel path tangential to thecylindrical region and corresponding to a first portion of theenvironment, and receiving a second input at a second face of the prismfor the rotational position of the cylindrical region, the second inputhaving a second travel path tangential to the cylindrical region andcorresponding to a second portion of the environment.
 24. The apparatusof claim 23, the instructions comprising further instructions capable ofdirecting a machine to perform: storing the first input and the secondinput for each of plural rotational positions of the cylindrical region;selecting a view point within the cylindrical region; and constructing aconvergent stereo image of the environment with respect to the selectedview point and the stored first and second inputs for the pluralrotational positions of the cylindrical region.
 25. The method of claim23, wherein the first travel path is opposite of the second travel path.26. The method of claim 23, wherein the first and second travel pathsare parallel to the rotation plane.
 27. An apparatus for acquiring inputfor a first image and a second image of a convergent stereo image pair,comprising: a deflector rotably mounted a distance from a rotation axis,the deflector having plural deflection regions; a receptor positionedproximate to the rotation axis, the receptor comprising a first portionof sensors and a second portion of sensors; a first memory for storing afirst input received at a first deflection region of the deflector anddeflected towards the first portion of sensors; and a second memory forstoring a second input received at a second deflection region anddeflected towards the second portion of sensors;
 28. The apparatus ofclaim 27, further comprising: an image constructor which determines thefirst image based at least in part on the first input, and the secondimage based at least in part on the second input.
 29. The apparatus ofclaim 28, further comprising: an interface for receiving a selected viewpoint; and a renderer for rendering a three dimensional imaged based atleast in part on the selected view point, the first image, and thesecond image.
 30. The apparatus of claim 27, wherein the deflectorrotates about the rotation axis, and while rotating, subsequent firstand second inputs are received, deflected, and stored in the firstmemory and the second memory.
 31. The apparatus of claim 27, furthercomprising: an interface for receiving a configuration input; andsetting the distance with respect to the configuration input.
 32. Theapparatus of claim 31, wherein the configuration input corresponds to aselected one of a desired depth of field for the convergent stereoimage, and a desired size for a region in which a viewpoint may beselected.